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ABSTRACT 


While it is widely aeeepted that polling can assess levels of popular support in a 
geographic area by surveying a cross-segment of its population, it is less well accepted 
that analysts can use social media analysis to assess sentiment or popular support within 
the same space. We examine this question by comparing geographically anchored polling 
and social media data, utilizing over 1.4 million geo-referenced messages sent through 
the Twitter network from Yemen over the period from October 2013 to January 2014, to 
assess both support for extremist groups and support to the Yemeni government. From 
our research, we conclude that social media data, when combined with polling, has a 
positive impact on analysis. It can also be a reliable source of stand-alone data for 
evaluating popular support under certain conditions. Therefore, we recommend future 
research projects focus on improving the quality of social media data and on operational 
changes to improve the integration of social media analysis into assessment plans. 
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I. 


INTRODUCTION 


Since nations have existed, they have faced population-eentric forms of warfare— 
rebellion, insurgency, and civil war—in which insurgents typieally battle entrenched 
governments for the loyalty of the populace, with each side’s movements and intentions 
obscured by the fog of conflict. In the modern era, governments rely on laborious, time- 
eonsuming, and expensive opinion polls to navigate through this fog. Opinion polls are 
like green and red marker buoys in a channel, spaeed intermittently along the way to help 
captains determine their positions. Like buoys, polling is reliable and sheds light on 
known positions in time, but polls are inherently spaced apart from one another and thus 
laek timeliness. In a sea covered by a thick bla nk et of fog, buoys ean easily be obscured. 
Like captains navigating in unclear conditions, military commanders navigating through 
the fog of eonflict require timely location information, or else their efforts may founder. 

Our research attempts to determine whether social media data analysis can serve 
as a proxy for or fill in the gap between opinion polls. If so, military commanders eould 
continuously monitor population sentiment and determine whether their efforts were 
affecting the population. 

To answer our questions, we developed a method that reliably measures the 
geographie distribution of aggregate popular sentiments using data derived from publie 
messages sent through the Twitter network. We then compared these measurements to 
the spatial distribution of sentiments derived from existing public opinion polls collected 
during the same period. 

For robustness, we tested our approaeh in Yemen, a diffieult test country. 
Compared to Western countries, Yemen has low levels of Internet access and soeial 
media users. The country is also beset by low levels of governance and violent extremist 
organizations battling for the loyalty of the populace. Yemen is typical of a country that 
the U.S. military would be tasked to deal with because it is a breeding ground for 
terrorism. 
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In total, the data used to develop our new metries eonsisted of over 1.4 million 
tweets, observed in Yemen over the period Oetober 2013 to January 2014. From our 
researeh we eonelude that soeial media data, when eombined with traditional polling 
approaehes, ean greatly improve our situational awareness over what ean be learned from 
polling alone. Under some eireumstanees, it may be possible to use soeial media data as a 
stand-alone tool for evaluating negative sentiment for a standing government or violent 
extremist organizations vying for eontrol of a eountry. However, we also find that our 
method performed better at identifying geographie patterns of negative feelings versus 
positive feelings regarding the government of Yemen and violent extremist organizations. 

We reeommend further researeh foeus on improving the algorithm that we 
developed. Speeifieally, we reeommend testing different timeframes in Yemen, different 
eountries, and Western eountries. Finally, we reeommend military eommanders make 
greater efforts to eombine the use of soeial media data and more traditional polling 
approaehes. Doing so will allow eommanders to better foeus eounterinsurgeney 
operations on the disaffeeted portions of the population where partnerships are most 
eritioal. 
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II. LITERATURE REVIEW 


A. MILITARY DOCTRINE AND ASSESSMENT 

This project was inspired in large part by the team’s combined 50 years of service 
in both the U.S. Navy and U.S. Army, working against asymmetric actors. Our 
experience waging counterinsurgency operations across the globe has taught us that 
sentiment is fundamental in gauging success in the type of warfare the U.S. has been 
focused on for decades.i Regrettably, we have watched numerous Special Forces 
commanders struggle to evaluate the strategic effectiveness of their counterinsurgency 
operations. We feel this is mainly because they did not have enough information on the 
populace in the regions where they held responsibility. Compounding this issue is the fact 
that U.S. Joint Doctrine Publications instruct commanders to measure the effectiveness of 
their operations as part of a feedback loop, but fail to provide clear ideas of how to do the 
job or resources to help .2 

A portion of the doctrine issue we illuminate may be due to the disagreement 
regarding whether to measure sentiment in the first place. Some would argue there is no 
difference between conventional military Clausewitzian force-on-force conflicts and a 
“people’s war” (e.g., low-intensity conflict, hybrid warfare, irregular warfare, etc.).34 
Therefore, the military should focus simply on the enemy, making popular support 
irrelevant.3 Other scholars and theorists, such as counterinsurgency warfare experts, 
would argue that only the actor who wins and maintains popular support will achieve his 

^ Seth G. Jones, RAND Counterinsurgency Study, vol. 4, Counterinsurgency in Afghanistan (Santa 
Moniea, CA: RAND Corporation, 2008), 7. 

2 Jaek D. Kern, “Assessment: Measures of Performanee and Measures of Effeetiveness,” Military 
Intelligence Professional Bulletin 35, no. 2 (2009): 48-50. 

3 Stuart Kinross, “Clausewitz and Low-Intensity Confliet,” Journal of Strategic Studies 27, no. 1 
(2004): 35-58; Frank Hoffman, “Hybrid Warfare and Challenges,” JFQ 52, no. 1 (2009); James Claney 
and Chuek Crossett, “Measuring Effeetiveness in Irregular Warfare,” Parameters 37, no. 2 (2007): 88. 

4 Clausewitz also spoke about the “passions of the people.” The passions of the people were a key part 
of his trinity of war. 

3 Raphael Cohen, “Just How Important Are ‘Hearts and Minds’ Anyway? Counterinsurgeney Goes to 
the Polls,” Journal of Strategic Studies 37, no. 4 (June/July 2014): 609-636; Russell Weigley, The 
American Way of War: A History of United States Military Strategy and Policy (Bloomington, IN: Indiana 
University Press, 1977). 
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or her objectives. This is believed because, in Roger Trinquier’s words, “the ability to 
exercise power over a populace depends on the tacit or explicit agreement of the 
population and in worst-case scenarios on their submission.”^ 

Moreover, among those who do not question the relevance of population-centric 
counterinsurgency warfare, there is disagreement about whether to study quantifiable 
metrics or qualitative “hearts and minds” metrics, similar to the broader divisions found 
across the social sciences.^ 

Military units that value behavior-based metrics, such as Psychological 
Operations, have developed an approach called effects-based operations, which focuses 
on observable behavior-based metrics to assess military effectiveness.^ They analyze 
variables like security, economic indicators, justice indicators, and governance 
indicators.^ To do so, they gather and assess public attitudes, beliefs, atmospherics, and 
opinions via three means: mass-media analysis, open-source analysis, and opinion 
polling. Passive observation of a local population and mass-media open-source analysis 
(one-to-many communication modes) provide qualitative insight regarding the attitudes, 
beliefs, and opinions of key demographics and elites respectively.!*’ 


^ George Casey Jr. et al., Assessing War: The Challenge of Measuring Success and Failure 
(Washington, DC: Georgetown University Press, 2015), 218. 

^ Robert B. Sotire, Measuring Performance and Effectiveness in Irregular Warfare: Preventing 
Dysfunctional Behavior (Newport, RI: Naval War College, 2009). 

^ Edward Smith, “Effects-Based Operations,” m Applying Network-Centric Warfare in Peace, Crisis 
and War (Washington, DC: DOD CCRP, 2002). 

^ Jason Campbell and Jeremy Shapiro, “Afghanistan Index: Tracking Variables of Reconstruction and 
Security in Post-9/11 Afghanistan,” Brookings Institution, October 28, 2008, 4-49; Ethan B. Kapstein, 
“Measuring Progress in Modem Warfare,” Survival 54, no. I (03/01, 2012): I37-I58; M. Treblicock and 
R. Daniels, Rule of Law Reform and Development: Charting the Fragile Path of Progress (Surrey, UK: 
Edward Elgar Publishing, 2008), 42; Craig Cohen, “Measuring Progress in Stabilization and 
Reconstraction,” United States Institute of Peace: Stabilization and Reconstmction Series (2003): 1-12; 
Michael Dziedzic, Barbara Sotirin, and John Agoglia, eds., “Measuring Progress in Conflict Environments 
(MPICE),” United States Institute of Peace, March 19, 2008. 

!*! Michael T. Flynn, Matthew F. Pottinger, and Paul D. Batchelor, Fixing Intel: A Blueprint for 
Making Intelligence Relevant in Afghanistan (Washington, DC: Center for a New American Security, 
2010). doi:ADA511613; John Urry, Mobilities (Cambridge, UK: Polity, 2007); Robert Steele, “Open 
Source Intelligence,” in Handbook of Intelligence Studies, edited by Loch K. Johnson (Oxford, UK: 
Routledge, 2007), 129-147. 
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Proponents of effects-based operations assess support to military operations 
through behavior-based indieators. This approaeh is eommonly ealled “hearts and 
minds,” beeause it measures attitudes, beliefs, and opinions.! i However, erities of effeets- 
based operations point out questionable relationships between observable behavior and 
popular support indieators. 12 

B. SOCIAL MEDIA ANALYSIS 

An alternative approaeh to assessment, not considered within doctrines, may be 
provided through social media analysis. Social scientists now have access to massive 
quantities of data sourced through public Internet and social media sites. The techniques 
used to comb through the hard drives packed with data have various titles, which can be 
organized under the broad heading of data mining}^ In general terms, data mining is a 
process of description and prediction that allows analysts to sort, analyze, and visualize 
data. 14 One area of knowledge that builds from data mining is deriving meaning from 
written text. This has been called sentiment analysis, opinion mining, and subjectivity 
analysis We refer to Georgios Paltoglou, a senior lecturer from the University of 
Wolverhampton, United Kingdom, who defines it as follows: 

Specifically, opinion mining addresses the problem of detecting, 
extracting, analyzing, and quantifying expressions of private states in 
written text in an automatic, computer-mediated fashion. Particular 
emphasis should be placed on the term “computer-mediated,” as the field 


11 David Galula, Counterinsurgency Warfare: Theory and Practice (Greenwood Publishing Group, 
2006); United States Joint Chiefs of Staff, Joint Publication Counterinsurgency (JP 3-24) (Washington, 
DC: U.S. Government Printing Office, 2013); Jordan Stem, Civil Military Operations and Military 
Information Support Operations Coordination: A Non-Kinetic Ballast for Disciplined Counterinsurgency 
Operations (Bethseda, MD: Small Wars Foundation, 2011). 

12 Christopher Paul and William Marcellino, “Dominating Duffer’s Domain: Lessons for the 2f‘- 
Century Information Operations Practitioner,” RAND Corporation, Santa Monica: CA, 2016. 

1^ Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth, “From Data Mining to Knowledge 
Discovery in Databases.” A/Magazine 17, no. 3 (1996): 37. 

14 Alexander Furnas, “Everything You Wanted to Know about Data Mining but Were Afraid to Ask,” 
Atlantic, April 3, 2012. 

1^ Bo Pang and Lillian Lee. “Opinion Mining and Sentiment Analysis,” Foundations and Trends in 
Information Retrieval 2, wo. 1-2 (2008): 1-135. 
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has a particular focus on designing, analyzing, and implementing software 
that performs the aforementioned analysis in an automatie manner.!^ 

There are many soeial media sourees for data mining, ineluding soeial networking 
(e.g., Faeebook), bookmarking (e.g., Pinterest), soeial news (e.g., Reddit), media sharing 
(e.g., YouTube), mieroblogging (e.g., Twitter), and various blogs/forums. Researchers 
from numerous fields are looking deeper into questions using big data from soeial media 
to study behavior in humans, polities, financial markets, and publie opinion, and 
movement patterns of groups to identify the spread of diseases, migration patterns of 
animals, patterns of behavior in humans, and eross-border routes (e.g., smuggling 


Georgios Paltoglou, “Sentiment Analysis in Social Media,” in Online Collective Action, Dynamics 
of the Crowd in Social Media, eds. N. Agarwal, M. Lim, and R. T. Wigard (Vienna, Austria: Springer 
International Publishing, 2013), 3-17. 

Alexander Pak and Patrick Paroubek. “Twitter as a Corpus for Sentiment Analysis and Opinion 
Mining,” in Proceedings of the Seventh Conference on International Language Resources and Evaluation 
(LREC ‘10), eds. Nicoletta Calzolari et al (Valletta, Malta: European Language Resources Distribution 
Agency, May 2010); Fayyad, Piatetsky-Shapiro, and Smyth, “From Data Mining to Knowledge 
Discovery”; Stefan Stieglitz and Linh Dang-Xuan, “Political Communication and Influence through 
Microblogging: An Empirical Analysis of Sentiment in Twitter Messages and Retweet Behavior,” in 
Proceedings of the 45th Hawaii International Conference on System Science, ed. Ralph FI. Sprague Jr. 
(Washington, DC: IEEE Computer Society, 2012), 3500-3509; Mike Thelwall, Kevan Buckley, and 
Georgios Paltoglou, “Sentiment in Twitter Events,” Journal of the American Society for Information 
Science and Technology 62, no. 2 (2011): 406-418; Xiaofeng Wang, Matthew S. Gerber, and Donald E. 
Brown, “Automatic Crime Prediction Using Events Extracted from Twitter Posts,” in SBP'12 Proceedings 
of the 5th International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction, 
eds. Shanchieh Jay Yang, Ariel M. Greenberg, and Mica Endsley (Berlin, Germany: Springer-Verlag 
Berlin, Heidelberg, 2012), 231-238. 
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routes).!^ Other areas of researeh inelude the relationship between soeial media and 
individual or group behavior, whieh ean potentially assess the impaet of online aetivism 
or a government’s publie relations campaign.i^ There are also efforts to understand soeial 
mobilization and eolleetive violenee, of partieular value in assessing a population through 
soeial movement theory.20 However, many of these studies have faeed substantial 
barriers. For instanee, most studies foeus on Western eountries that use the English 
language and are dominated by Western culture.21 


Morgan R. Frank et al., “Ftappiness and the Patterns of Life: A Study of Geolocated Tweets,” 
Scientific Reports 3 (2013); Bartosz Flawelka et al., “Geo-Located Twitter as Proxy for Global Mobility 
Patterns,” Cartography and Geographic Information Science 41, no. 3 (2014): 260-271; Marcel Salathe et 
al., “Influenza A (H7N9) and the Importance of Digital Epidemiology,” New England Journal of Medicine 
369, no. 5 (Aug 1, 2013): 401-4; Justine 1. Blanford et al., “Geo-Located Tweets: Enhancing Mobility 
Maps and Capturing Cross-Border Movement,” PloS ONE 10, no. 6 (Jun 2015, 2015); Zhiyuan Cheng, 
James Caverlee, and Kyumin Lee, “You Are Where You Tweet: A Content-Based Approach to Geo- 
Locating Twitter Users,” in Proceedings of the 19th ACM International Conference on Information and 
Knowledge Management, 2010, 759-768; Ryan Compton, David Jurgens, and David Allen, “Geotagging 
One Hundred Million Twitter Accounts with Total Variation Minimization,” Big Data (Big Data), 2014 
IEEE International Conference on, 2014, 393-401; Michael D. Conover et al., “The Geospatial 
Characteristics of a Social Movement Communication Network,” PloS ONE 8, no. 3 (2013): e55957; 
Andreas Kaltenbmnner et al., “Far from the Eyes, Close on the Web: Impact of Geographic Distance on 
Online Social Interactions,” Proceedings of the 2012 ACM Workshop on Online Social Networks (New 
York: ACM, 2012), 19-24; Juhi Kulshrestha et al., “Geographic Dissection of the Twitter Network,” 
ICWSM, 2012; Ryong Lee and Kazutoshi Sumiya, “Measuring Geographical Regularities of Crowd 
Behaviors for Twitter-Based Geo-Social Event Detection,” in Proceedings of the 2nd ACM SIGSPATIAL 
International Workshop on Location-Based Social Networks (New York: ACM, 2010), 1-10; Kalev 
Leetam et al., “Mapping the Global Twitter Heartbeat: The Geography of Twitter,” First Monday 18, no. 5 
(2013); Lewis Mitchell et al., “The Geography of Happiness: Connecting Twitter Sentiment and 
Expression, Demographics, and Objective Characteristics of Place,” PloS ONE 8, no. 5 (2013): e64417; 
Stephen C. Nemeth, Jacob A. Mauslein, and Craig Stapley, “The Primacy of the Local: Identifying 
Terrorist Hot Spots Using Geographic Information Systems,” Journal of Politics 76, no. 2 (2014): 304- 
317; Yuri Takhteyev, Anatoliy Gruzd, and Barry Wellman, “Geography of Twitter Networks,” Social 
Networks 34, no. 1 (2012): 73-81; Quan Yuan et al., “Who, Where, When and What: Discover Spatio- 
Temporal Topics for Twitter Users,” in Proceedings of the 19th ACM SIGKDD International Conference 
on Knowledge Discovery and Data Mining, eds. Ingerjit S. Dhillon et al. (New York: ACM, 2013), 605- 
613. 

Raz Schwartz and Germaine R. Halegoua, “The Spatial Self: Location-Based Identity Performance 
on Social Media,” New MeJia & Society, April 9, 2014, 1-18. doi:10.1177/1461444814531364. 

20 Doowan Lee, "A Social Movement Approach to Unconventional Warfare," Special Warfare July- 
September, Volume 26, no. Issue 3 (2013), 27-32.; Doug McAdam, Political Process and the Development 
of Black Insurgency, 1930-1970, ed. 2ndUniversity of Chicago Press, LTD., London, 1982). 

21 T. Camber Warren, “Mapping the Rhetoric of Violence: Political Conflict Discourse and the 
Emergence of Identity Radicalization in Nigerian Social Media,” prepared for presentation at the Annual 
Meeting of the American Political Science Association, Sept. 3-6, 2015, San Francisco, CA. 
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Nevertheless, large strides have been made in analyzing individual messages to 
determine underlying sentiment, whieh is the focus of our research.22 These strides in 
sentiment studies have resulted in new approaches, including natural language 

processing, text analysis, linguistic inquiry, word count, keyword spotting, lexical 

23 

affinity, statistical methods, and concept-level techniques. Keyword spotting is the 

most simplistic approach, because it sorts text by category using unambiguous emotional 

24 

words. For example, if keyword software is programmed to look for the word 
government, this is because the word government has already been associated with an 
emotional category like good or bad. Another approach is lexical affinity, which sorts 
text much the same way as keyword spotting. The big difference is that it is able to 


22 Ahmed Abbasi, Hsinchuti Chen, and Arab Salem, “Sentiment Analysis in Multiple Languages: 
Feature Seleetion for Opinion Classifieation in Web Forums,” ACM Transactions on Information Systems 
(TOIS) 26, no. 3 (2008): 12; Apoorv Agarwal et al., “Sentiment Analysis of Twitter Data,” in Proceedings 
of the Workshop on Languages in Social Media, eds. Meena Nagarajan and Miehael Gamon, Assoeiation 
for Computational Linguisties, 2011, 30-38; Younggue Bae and Flongehul Lee, “Sentiment Analysis of 
Twitter Audienees: Measuring the Positive or Negative Influenee of Popular Twitterers,” Journal of the 
American Society for Information Science and Technology 63, no. 12 (2012): 2521-2535; Albert Bifet and 
Elbe Frank, “Sentiment Knowledge Diseovery in Twitter Streaming Data,” in DS’lO Proceedings of the 
13'^ International Conference on Discovery Science, eds. Berhamd Pfahringer, Geoff Flolmes, and Aehim 
Floffmann (Heidelberg, Germany: Springer-Verlag, 2010), 1-15; Johan Bollen, Alberto Pepe, and Huina 
Mao, “Modeling Pubbe Mood and Emotion: Twitter Sentiment and Soeio-Eeonomie Phenomena,” in 
Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM 2011), eds. 
N. Nieolov and J. G. Shanahan (Barcelona, Spain: AAAI Press, July 2011), 1-10; Peter Sheridan Dodds et 
al., “Temporal Patterns of Happiness and Information in a Global Social Network: Hedonometrics and 
Twitter,” PloS ONE 6, no. 12 (2011): e26752; Rui Fan et al., “Anger Is More Influential than Joy: 

Sentiment Correlation in Weibo.” PloS ONE 9, no. 10 (2014): el 10184; M. Ghiassi, J. Skinner, and D. 
Zimbra, “Twitter Brand Sentiment Analysis: A Hybrid System Using N-Gram Analysis and Dynamic 
Artificial Neural Network,” Expert Systems with Applications 40, no. 16 (2013): 6266-6282; Scott A 
Golder and Michael W. Macy. “Diurnal and Seasonal Mood Vary with Work, Sleep, and Daylength across 
Diverse Cultures,” Science 333, no. 6051 (2011): 1878-1881; Shu Huang et al., “Sentiment and Topic 
Analysis on Social Media: A Multi-Task Multi-Label Classification Approach,” in Proceedings of the 5th 
Annual ACM Web Science Conference (New York: ACM, 2013), 172-181; Long Jiang et al., “Target- 
Dependent Twitter Sentiment Classification,” in Proceedings of the 49th Annual Meeting of the Association 
for Computational Linguistics: Human Language Technologies, vol. 1 (Stroudsburg, PA: Association for 
Computational Linguistics, 2011), 151-160; Lewis Mitchell et al., “The Geography of Happiness”; 

Brendan O’Connor et al., “From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series,” 
in Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, 2010, 1-2. 

23 Erik Cambria et al., “New Avenues in Opinion Mining and Sentiment Analysis,” IEEE Intelligent 
Systems no. 2 (2013): 15-21; Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan, “Thumbs Up? 
Sentiment Classification Using Machine Learning Techniques,” in Proceedings of the 2002 Conference on 
Empirical Methods in Natural Language Processing (Stroudsburg, PA: Association for Computational 
Linguistics, 2002); Namrata Godbole, Manja Srinivasaiah, and Steven Skiena, “Large-Scale Sentiment 
Analysis for News and Blogs,” Icwsm 7, no. 21 (2007): 219-222. 

24 Andrew Ortony, Gerald L. Clore, and Allan Collins, The Cognitive Structure of Emotions (New 
York: Cambridge University Press, 1990). 
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measure the distanee between words in a text string and assign an intensity value based 
on the distanee. 

The statistieal method is slightly more eomplex. It uses natural language 
proeessing (NLP) to cull through massive amounts of data and assigns values to patterns 
in the text. One example is the “bag of words” approach, which analyzes a text without 
considering the order or distance between words. This approach essentially counts the 
frequency of words co-occurrences, and then infers “negative” or “positive” valence 
categories based upon the patterns of these co-occurrences. It is commonly used to filter 
spam by using a list of valued words to determine whether an email is an advertisement 

(i.e., negative words) or a legitimate communication (i.e., positive words).^^ 

The most complex approaches are known as concept-level techniques. These 

techniques primarily focus on semantic analysis of the text. This means that they evaluate 

27 

how concepts in one text relate to concepts in another text. This form of social media 
analysis, rooted in lexical analysis, examines the content of messages.^^ Lexical 
analysis’s subfields and analytical methods include discourse analysis, narrative analysis, 
trend analysis, and sentiment analysis. Discourse analysis reviews both two-way 
conversations and one-way rhetoric. These inputs typically flow from an organization’s 
spokesperson or a key communicator to his or her listeners, readers, or viewers.29 
Narrative analysis highlights how an individual or group reflects their understanding of 
their place in the world; this approach is increasingly applied within the field of social 


Hugo Liu, Henry Lieberman, and Ted Selker, “A Model of Textual Affeet Sensing Using Real- 
World Knowledge,” in Proceedings of the 2003 International Conference on Intelligent User Interfaces 
(New York: ACM, 2003). 

Peter Turney, “Thumbs Up or Thumbs Down? Semantie Orientation Applied to Unsupervised 
Classifieation of Reviews,” in Proceedings of the 40th Annual Meeting of the Association for 
Computational Linguistics (Stroudsburg, PA: Assoeiation for Computational Linguisties, 2002), 417-424. 

Erik Cambria et ah, “Sentie Patterns: Dependeney-Based Rules for Coneept-Level Sentiment 
Analysis,” Knowledge-Based Systems 69, (2014): 45-63. 

Pang and Lee, “Opinion Mining and Sentiment Analysis.” 

Teun A. van Dijk, “Critieal Diseourse Analysis,” in The Handbook of Discourse Analysis, eds. 
Deborah Sehiffrin, Deborah Tannen, and Heidi E. Hamilton (Malden, Massaehusetts: Blaekwell Publishers, 
2001, 349-371. 
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movement theory Trend analysis instead foeuses on temporal patterns in diseourse, 
mathematieally identifies items experieneing aeeelerating inereases in frequeney, whether 
those items are word and phrases used in seareh engines, or ‘trending’ stories on news 
sites .31 


The output of sentiment analysis ean take multiple forms: ineluding ordered, 
eontinuous, and eategorieal values. Ordered outputs might be either positive or negative, 
or might include a ternary division between positive, negative, and neutral.32 Categorical 
values are similar, but they assign no actual numerical content, so the categories are just 
boxes like “a” or “cat,” which do not imply that one value is numerically greater than 
another. 

Researchers have used numerous methods to analyze and visually represent social 
media data. Geospatial analysis, for instance, is a visual representation of data that looks 
at identifying spatially related patterns between events and/or individuals and their 
locations.33 A great example of this approach is heat maps. A heat map is a graphical 
representation of data in which the individual values in a matrix are represented as colors. 
The color coding highlights distinctions between values taken by variables in a hierarchy. 


30 Catherine K. Riessman, “Narrative Analysis,” in M.S. Lewis-Beck, A. Bryman, and T. Luting Liao, 
eds., SAGE Encyclopedia of Social Science Research Methods, vol. 3 (Thousand Oaks, CA: SAGE 
Publications, 2003); Arthur A. Berger, Media Analysis Techniques (Thousand Oaks, CA: SAGE 
Publications, 2013); Ruth E. Page, Stories and Social Media: Identities and Interaction (Oxford, UK: 
Routledge, 2013). 

31 Jey H. Lau, Nigel Collier, and Timothy Baldwin. “On-Line Trend Analysis with Topic Models: 
#Twitter Trends Detection Topic Model Online,” conference paper. Proceedings ofCOLING 2012, 
December 2012; M. Osborne et ah, “Bieber No More: First Story Detection Using Twitter and Wikipedia,” 
in SIGIR 2012 Workshop on Time-Aware Information Access Oregon, 2012; S. Petrovic, M. Osborne, and 
V. Lavrenko, “Streaming First Story Detection with Application to Twitter,” in Proceedings of Human 
Language Technologies: The 11th Annual Conference of the North American Chapter of the Association 
for Computational Linguistics (NAACL HLT 2010) (Stroudsburg, PA: Association for Computational 
Linguistics: 2010), 181-189. 

32 Anna Chmiel et ah, “Negative Emotions Boost User Activity at BBC Forum,” Physica A: Statistical 
Mechanics and Its Applications 390, no. 16 (2011): 2936-2944; Marija Mitrovic, Georgios Paltoglou, and 
Bosiljka Tadic, “Quantitative Analysis of Bloggers’ Collective Behavior Powered by Emotions,” Journal 
of Statistical Mechanics: Theory and Experiment 2011, no. 02 (2011): P02005. 

33 Anthony Stefanidis, Andrew Crooks, and Jacek Radzikowski, “Flarvesting Ambient Geospatial 
Information from Social Media Feeds,” Geojournal 78, no. 2 (2013): 319-338; Daniel Sui and Michael 
Goodchild, “The Convergence of GIS and Social Media: Challenges for GIScience,” International Journal 
of Geographical Information Science 25, no. 11 (2011): 1737-1748; Sarah Elwood, Michael F. Goodchild, 
and Daniel Z. Sui, “Researching Volunteered Geographic Information: Spatial Data, Geographic Research, 
and New Social Practice,” Annals of the Association of American Geographers 102, no. 3 (2012): 571-590. 
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Social network analysis (SNA), on the other hand, uses soeial media data to identify the 
ties between aetors and the strengths of those ties.^^ The ties are frequently displayed in a 
hub-and-spoke wheel of aetors and their eonneetions to other aetors in a network. 

A variety of issues arise when attempting to determine sentiment using the 
Internet and soeial media, ineluding the source, sample representation, and language. It is 
important to note that data derived from these sourees often do not represent proper 
“samples” in the statistieal sense.^5 Souree issues involve not knowing where the data 
originates from. Imagine a pollster asking questions of a person who is standing behind a 
white sheet. The pollster eannot know for eertain the identity of the souree. Data found 
online may have all kinds of unwanted sourees ineluding hots, Internet agents, wanderers, 
brokers, trolls, propagandists, and even offieial news outlets that can skew the results.36 
Additionally, in demographic terms, “Internet and soeial media users are not likely a true 
representation of a given population,” and as a result they do not necessarily provide a 
representative sample for anafysis.37 For instanee, aeeess to the Internet in many eountries 
may only be found in urban areas with improved infrastrueture. People from rural, less 
connected areas may therefore be underrepresented. Other demographie issues related to 
Internet and soeial media aeeess includes age, education, gender, and ineome. Still, there 
are widely aeeepted statistieal teehniques for dealing with sueh issues. As we explain 
more fully, it may be possible reduee such difficulties by using ‘eontrol variables’ to hold 
eertain faetors eonstant when examining the effeets of a variable of interest. 


34 Sean F. Everton, Disrupting Dark Networks, vof 34 (New York: Cambridge University Press, 
2012); Stuart Kosehade, “A Soeial Network Analysis of Jemaah Islamiyah: The Applieations to 
Counterterrorism and Intelligenee,” Studies in Conflict & Terrorism 29, no. 6 (2006): 559-575; Stanley 
Wasserman and Katherine Faust, Social Network Analysis: Methods and Applications, vol. 8 (New York: 
Cambridge University Press, 1994). 

33 Stephen Ansolabehere and Eitan Flersh, “Validation: What Big Data Reveal about Survey 
Misreporting and the Real Eleetorate,” Political Analysis (2012): mps023; Pablo Barbera, “Flow Soeial 
Media Reduees Mass Politieal Polarization: Evidenee from Germany, Spain, and the US,” working paper 
prepared for the 2015 APSA Conferenee, New York University, 2014; Alan Mislove et ah, “Understanding 
the Demographies of Twitter Users,” ICWSM 11 (2011): 5. 

36 Fah-Chun Cheong, Internet Agents: Spiders, Wanderers, Brokers, and Bots (San Franeiseo, CA: 
New Riders Publishing, 1996). 

3^ Ansolabehere and Flersh, “Validation: What Big Data Reveal”; Pablo Barbera and Gonzalo Rivero, 
“Understanding the Politieal Representativeness of Twitter Users,” Social Science Computer Review 33, 
no. 6 (Deeember 1, 2015): 712-729; Mislove et ah, “Understanding the Demographies of Twitter Users.” 
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Since most social networking platforms were built by English speakers, the 
diversity of written formats seen in other languages can pose its own challenges. 
Processing English text is relatively easy compared to processing text in some foreign 
languages. English uses a relatively simple alphabet that contains separate letters for 
consonants and vowels. Other writing forms, for instance Chinese logograms, are built of 
glyphs that represent words. A glyph is a meaningful component of a word, rather than a 
phonetic element. Ideographic writing is based on pictures that represent concepts or 
ideas, rather than specific words. In contrast, a segmental script, like Arabic, uses 
graphemes that represent phonemes, distinct units of sound that distinguish one word 
from another. 3 8 

Given these difficulties, one approach, taken by many existing works, is to stick 
with English-language databases. However, when faced with a research question that 
requires analysis of foreign societies, such an approach seems too sharply limiting. 
Promisingly, recent advances in machine translation have increased the ease of 
generating relatively reliable translations of single words and short phrases.39 Por 
instance, researchers can access automated machine-generated translations through online 
tools, such as the Google Translate API. This service allows for simple translation from a 
wide variety of languages to other languages. More nuanced approaches to machine 
translation utilize natural language parsers to consider the broader grammatical structures 
of sentences and paragraphs. However, this requires careful tokenization of the text 
combined with language-specific grammar rules, which can make this approach cost- 
prohibitive in computational terms when face with very large textual corpora. 


38 Michael A. Halliday, Spoken and Written Language (Oxford, UK: Oxford University Press, 1989). 

39 Mohammad Salameh, Saif M. Mohammad, and Svetlana Kiritchenko, "Sentiment after Translation: 
A Case-Study on Arabic Social Media Posts," in Proceedings of the 2015 Conference of the North 
American Chapter of the Association for Computational Linguistics: Human Language Technologies 
(2015): 767-777. 
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III. BACKGROUND—YEMEN 


Yemen is an ideal plaee to test our researeh. Compared to Western eountries, 
Yemen has low levels of Internet aeeess and few soeial media users. The eountry is also 
struggling with low levels of govemanee and violent extremist organizations battling for 
the loyalty of the populaee. Yemen is typieal of a country that the U.S. military would be 
tasked to deal with, because it is a breeding ground for terrorism. The following section 
delves into the problems facing Yemen. 

For the majority of the 20th century, Yemen was divided between the Ottoman 
and British empires. Following the fall of the Ottoman Empire, northern Yemen became 
the Yemen Arab Republic. Southern Yemen (People’s Democratic Republic of Yemen) 
remained a British colony until 1967. Finally, in 1990, northern and southern Yemen 
united. Four years after becoming a united nation, Yemen experienced a two-month-long 
civil war over governance. 

Since the 1994 civil war, Yemen has become increasingly unstable. Violent 
extremist organizations have filled gaps where the government has failed. According to 
the Fund for Peace, the country was ranked in the top 10 most fragile in both 2013 and 
2014.40 Numerous organizations are vying for control of the weak Yemeni government. 

A. VIOLENT EXTREMIST ORGANIZATIONS 

Violent extremist organizations (VEO) in Yemen include the Shia Houthis, Sunni 
Salafi, and al-Qaeda in the Arabian Peninsula (AQAP). All three VEOs have challenged 
the government over control of territory, threatened government progress initiatives such 
as the National Dialogue Conference (NDC), conducted kidnapped for ransom, murdered 
civilians, and leveraged poor socioeconomic conditions and government corruption, 
leaving Yemen unstable and a security concern for the rest of the world and organizations 
that want to help. Non-governmental agencies are loath to help for fear of their safety on 
the ground. Eor example, on November 14, 2013, VEOs sabotaged the Internet and cell 

40 Fund for Peace, “The Fragile States Index 2015,” accessed June 15, 2016, 
http: //fsi. fundforpeac e. org/rankings -2014. 
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phone towers in the Shabwa Governorate The Yemeni population suffered as a result, 
and the government’s response was popularly viewed as insuffieient. The government 
elaimed many exeuses for their lack of action, including insufficient resources, security 
concerns, and roads barricaded by the VEOs. Ultimately, the government missed the 
opportunity to exploit VEO violent actions via propaganda or other mean, leaving the 
VEOs impervious to public scrutiny, despite their attacks on public infrastructure. 

B. HOUTHIS 

The Shia Houthis believe they are direct descendants of the Prophet Muhammad, 
making them the lawful rulers of Yemen.42 Needless to say, this places them at odds with 
the government. Because of their violent actions, they are deemed insurgents. 
Compounding the discontent stemming from the Houthis’ religious justification, the 
government has intentionally marginalized the group, placing them at odds with the rest 
of society and reducing their socioeconomic significance. Marginalization of the Houthis 
began with Saleh, president of Yemen from 1990 to 2012, and is believed to be the 
foundation for the 2004 Yemen civil war.43 

Saudi Arabia has generally viewed the Houthis as a threat and readily agreed to 
the Yemeni government’s request to bomb the Houthis’ home territory along the Saudi 
border in November 2009. Additionally, Saudi Arabia is claimed to have contributed 
between $200 and $300 million annually in security assistance to President Saleh.'^^ 
Saudi Arabia was also involved in exerting Wahhabi influence in Yemen and most 
recently in the northern Sa’ada Governorate."^^ The Houthis remain wary of attacks from 
Saudi Arabia and the Yemeni Army. 


41 Ibid. 

42 U.S. Government, U.S. Military, and Department of Defense, Yemen in Perspective - Orientation 
Guide and Yemeni Cultural Orientation: Geography, History, Economy, Security, Customs, Aden, Sanaa, 
Sunni, and Shi’a, Sufism, Qat Chewing, Al-Qaeda, Houthi Rebellion (Cleveland, OH: Progressive 
Management, 2015), 1-658, Kindle edition. 

43 Ibid. 

44 Ibid. 

45 Ibid. 
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Iran, on the other hand, generally views the Houthis as useful proxies against the 
Saudis. Interestingly, Iran and the Houthis do not follow the same Shia ideology. Iran 
eonveniently ignores the faet that the Houthis follow Fiver Shia, which does not 
acknowledge later imams, while the Iranians believe in Twelver Shia. Despite this 
religious difference, Iran is suspected of providing financial funding, military training, 
and material aid to the Houthis via Hezbollah.46 

C. SALAFIS 

Sunni Salafism was established around the middle of the 19th century in Egypt.47 
Salafi Muslims moved to Yemen about three decades ago."^^ They believe in re¬ 
establishing ancient traditions of Islam and do not believe in religious innovation.49 The 
movement is divided into three categories: those who avoid politics, those who advocate 
in favor of politics, and those who form small groups of jihadists.^*’ Salafism is often 
associated with Wahhabism, even though the two ideologies are dissimilar.^i 

In the fall of 2011, Salafis began smuggling weapons into the town of Dammaj, 
despite Houthi requests to stop. This resulted in two years of infighting between Sunni 
and Shia Muslims in Yemen. The Houthis and Salafis perpetuated the conflict until a 
presidential committee brokered a ceasefire on January 11, 2014. Part of the deal 
involved the forced relocation of hundreds of non-local Salafi families, in addition to 
150,000 refugees from the conflict area.52 

U.S. Government, Department of Defense, and U.S. Army, The Conflicts in Yemen and U.S. 
National Security - Yemeni Regional Politics and Saudi Arabia, Drones, Qat Chewing, Al-Qaeda, War on 
Terror, Houthi Tribesmen Rebellion, Zaydi Shiite Sect, Kleptocracy (Cleveland, OH: Progressive 
Management, 2014), Kindle edition. 

Joseph J. Hobbs, Fundamentals of World Regional Geography, ed. 4, (Cengage Learning, 2016), 

232. 

U.S. Government, Department of Defense, U.S. Army, U.S. Navy, U.S. Marine Corps, Army War 
College, and U.S. State Department, 2015 Complete Guide to Al-Qaeda in Yemen: Al-Qaida in the Arabian 
Peninsula (AQAP) (Cleveland, OH: Progressive Management, 2015), Kindle edition. 

"Politics and the Puritanical,” Economist. June 2015, "http://www.economist.com/news/middle- 
east-and-africa/21656189-islams-most-conservative-adherents-are-finding-politics-hard-it-beats. 

50 Ibid. 

51 Ibid. 

52 U.S. Government et al.. The Conflicts in Yemen and U.S. National Security, 232. 
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D. AL-QAEDA IN THE ARABIAN PENINSULA AND YEMEN 

AQAP has systematically built their network in Yemen over time, starting in 2006. 
By 2009, AQAP was known simply as Al-Qaeda in Yemen (AQY) and had grown strong 
enough to announce a southern independence movement. Bin Laden noticed that tribal 
support was imperative, and in 2000 sent messengers to Yemen to win local support 
through propaganda, welfare programs, and other essential material needed in 
impoverished villages.^3 

AQY followed a systematic process for every village they entered. They began by 
announcing their arrival by disseminating propaganda handbills, CDs, and DVDs which 
contained their dawa ideology. Slowly, they set about convincing influence leaders in 
each village to support them. Once they had influenced a small number of well-respected 
villagers to vouch for them, they would leave a few fighters behind and move on to the 
next village.In a relatively short time, they had exponentially increased domestic 
support of AQY and global support of AQAP. 

AQY also used other relatively ingenious methods to expand in Yemen, such as 
marrying into a given village and planning prison breaks for members who were 
imprisoned. Ultimately, they took advantage of uneducated, impoverished, and 
disenfranchised Yeminis villages. 

They also had a well-developed communication and marketing strategy that 
enabled them to build connective insurgent tissue in Yemen and around the globe. Their 
marketing campaign exponentially increased domestic support of AQY in impoverished 
and disenfranchised Yemini villages. 


Abdel Bari Atwan, After Bin Laden: Al-Qaeda, the Next Generation (London, UK: Saqi Books, 
London, 2012), 303. 

54 Ibid. 

55 Ibid. 

56 Ibid. 
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AQY fighters also assisted villagers with basie seeurity. Villagers in the Rada’a 
distriet, A1 Bayda Governorate, were seeking poliee or military basie seeurity, whieh they 
were not reeeiving from the national govemment.57 

Of eourse, not everything AQY did was positive. On Deeember 5, 2013, AQY 
attaeked a hospital, leaving over 50 people dead and 215 injured. Seeing the negative 
reaetion, AQY sent an apology to the families of the deeeased but stated that though the 
attaek had been an aeeident, eompartments in the building had supported U.S. drone 
operations. 

E. GOVERNMENT OF YEMEN 

President Had! (Yemeni president February 2012-present) has generally been 
assoeiated with attempts to transform Yemen and heal its deep wounds. His greatest 
aehievement may be the National Dialogue Conferenee (NDC), whieh eonvened in 
November 2013 and eoneluded in January 2014. Had! began the eonferenee by returning 
land in southern Yemen seized during the eivil war.^^ During the eonferenee, he proposed 
transforming Yemen into a federal system with six regions.^*’ Had! believed having six 
regions would benefit the southern Yemeni eities, his base of support. This plan did 
nothing for the opposition, however. Houthis living in the poorest northern mountain 
region of Yemen would reeeive minimal finaneial support and lose their sea aeeess. Not 
surprisingly, Hadi was unable to gain the support he needed to enaet his federation 
proposal, beeause the Houthis saw no benefit in it.^i 

In a strategie sense, the NDC was an initiative to bring peaee to Yemen, yet it was 
immediately marked with violenee. The eonferenee started with an assassination of a 


Ibid. 

Ibid. 

Ibid. 


U.S. Government et al., 2015 Complete Guide to Al-Qaeda in Yemen. 

“Yemen Crisis: Who Is Fighting Whom?” BBC News, October 14, 2016, http://www.bbc.com/ 
news/world-middIe-east-29319423. 
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Houthi representative on November 22, 2013.^2 jt was extended two weeks past its 
seheduled conclusion in January because of an impasse and then completely collapsed 
with the assassination of a second Houthi representative on January 21, 2014. 

Internationally, Yemen was having mixed luck, though the country was making 
progress in some circles. In October 2013, Hadi met with EU leaders from France and 
Germany who praised the NDC initiative.^3 Qn the other hand. Transparency 
International ranked Yemen in the top 10 most corrupt states. 

Domestically, the Yemeni government faces serious governance challenges. The 
government approved drone strikes and army attacks on VEOs, which were likely the 
cause of the numerous antigovernment protests that followed. Citizens then began 
demanding human rights after AQY enacted sharia law in several areas they had siezed. 
Regions and tribes that were unhappy with sharia law went to the NDC to express their 
views. People unhappy with police corruption protested with slogans such as “Handcuffs 
Must Break.”65 Jq make matters worse, Yemen approved several U.S. drones strikes that 
resulted in collateral damage. The “Red Wedding” was one such drone strike. On 
December 12, 2013, 15 people were killed on their way to a wedding in A1 Bay da. 
Tragically, the wedding party had been mistaken for an AQAP convoy.66 Yemen’s 
government also faced issues providing basic necessities such as electricity, low energy 
and food prices, and security. The International Institute for Strategic Studies Armed 
Conflict reported on January 6, 2014, that 54% of Yemenis were living in poverty and 
40% of Yemenis were unemployed.67 Roads and travel inside much of the country have 
been blocked by VEOs. 

62 International Institute for Strategic Studies, “Yemen (Houthis / AQAP / SMM),” Armed Conflict 
Database, accessed October 11, 2016, http://acd.iiss.org/en/conflicts/yemen—houthis-aqap-smm- 

9651 ?year=2013 &month= 10. 

63 Ibid. 

64 Transparency International, “Corruption Perceptions Index 2013,” accessed June 14, 2016, 
http://www.transparency.org/cpi2013/results. 

65 Ibid. 

66 Ibid. 

62 International Institute for Strategic Studies, “Yemen (Houthis / AQAP / SMM),” Armed Conflict 
Database, accessed October 11, 2016, http://acd.iiss.org/en/conflicts/yemen—houthis-aqap-smm- 
9651 ?year=2013 &month= 10 
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IV. RESEARCH METHODS 


A. HYPOTHESIS 

We propose that soeial media analysis can overcome the limitations of traditional 
collection and assessment methods by assessing a population’s attitudes, beliefs, and 
opinions using social media data and sentiment analysis. If relationship is found via 
logistical regression analysis, the study may provide scholars, practitioners, and the U.S. 
military a new method to analyze sentiment in lieu of a traditional survey and a way to 
assess the full spectrum of government operations and lines of effort in conflict zones. 
Additionally, future research may provide insight into social network analysis, social 
movement theory, and communications theory. 

B. DATA AND METHODS 

1. Social Media 

We began our research with an archive of Twitter data that contained three billion 
messages and 40 terabytes of data in total, licensed by NPS. The archive represented a 
10% randomized sample of the total number of tweets that occurred over a 12-month 
timeframe. Our research analyzed data spanning a four-month timeframe, from October 
1, 2013, through January 31, 2014. 

Traditional computational approaches search across strings of text in serial order. 
However, searching 40 terabytes of Twitter traffic using standard computational 
approaches would be very cumbersome and slow. Our approach instead utilized a highly 
parallelized in-memory database application developed by Camber Warren to speed the 
computations.^^ The approach utilizes “swarms” of parallel computational units to 
operate in tandem. This requires parallel loading of individual CPU cores, allowing it to 


T. Camber Warren, “Mapping the Rhetorie of Violenee: Politieal Confliet Discourse and the 
Emergence of Identity Radicalization in Nigerian Social Media,” prepared for presentation at the Annual 
Meeting of the American Political Science Association, September 3-6, 2015, San Francisco, CA. 
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avoid “resource conflicts, without the need for hierarchical control structures,” which are 
a traditional computational bottleneck.^^ 

The first task using this approach is to reference each message to a location in 
time and space. Each tweet contains underlying metadata that identifies the tweet’s date 
time stamp and the user’s location. The user location is based on either the user’s profile 
setting for their hometown or, less commonly, geo-tagging on the tweet itself The 
location is converted into latitude/longitude, allowing us to match it to other geo- 
referenced data sources. 

2. Sentiment Dictionary 

Our next task was to develop the parameters of our message content search, by 
building a “sentiment dictionary” (sometimes called a “sentiment lexicon”). This is a list 
of words that are assigned positive, negative, and/or neutral values manually or through 
natural language processing (NLP). We chose the AFINN dictionary that was developed 
by Finn Arup Nielsen.^o Qne advantage of this particular lexicon is that it explicitly 
captures intensity, using a scale that ranges from -5 to +5, allowing researchers to 
eliminate neutral responses and focus on extreme negative or extreme positive responses. 
We limited our lexicon to words with score magnitudes of at least 3 (positive or negative) 
in order to reduce neutral responses and limit analysis to the far left and right sides of the 
response scale. 

We also selected concept search terms reflecting key aspects of Yemeni society 
and politics. First, we sought to capture the concept of institutional confidence. The 
naming convention for search terms associated with government support was designed to 
include formal and informal ways of referencing government institutions: 

1) Yemeni + name of institution/individual (e.g., Yemeni Army, Yemeni Soldiers), 

2) our + name of institution/individual (e.g., our government, our president). 


69 Ibid. 

Finn Arup Nielsen, “A new ANEW: Evaluation of a word list for sentiment analysis in 
microblogs,” arXiv preprint arXiv: 1103.2903 (2011). 
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3) my + name of institution/individual (e.g., my city council, my mayor), 

4) location + name of institution/individual (e.g., Aden’s City Council, Aden’s mayor), 

5) title + name of senior individual associated with an institution (e.g.. President Hadi), 

6) name of senior individual associated with an institution without title (e.g., Abdullah 
Mohsen al-Akwa). 

Second, we sought to capture the concept of support for extremism. The naming 
convention for search terms associated with extremist support included the different ways 
the organization was identified along with senior leaders within the organization. The 
names and leaders were derived from government websites and multiple conflict 
databases. A list of search terms was compiled to separately assess each of the 10 
institutions and the violent extremist organizations evaluated in the survey. The 
combination of all terms associated with the government comprised the search criteria for 
institution. All terms associated with VEOs, along with specific terms related to jihad and 
sharia, comprised the search criteria for extremism. 

Both the sentiment dictionary and concept dictionary were then translated from 
English to Arabic using the automated Google Translate API. In an attempt to address the 
impact of translation potentially switching a word’s sentiment, we also translated each 
word back into English and replaced words that no longer matched their originals with 
manually selected words from Google Translate or else deleted them. We checked the 
Google Translate results with two humans. The first was Major Joshua Wines, and the 
second was a native Arabic speaker who teaches Arabic at the U.S. Defense Eanguage 
Institute in Monterey, California. 

3. Kernel Density Estimates 

Traditional spatial regression techniques entail selecting a political boundary like 
a city, county, or state, as the unit of analysis. However, such approaches are subject to a 


Concept dictionary is in attached Appendix: Concept Dictionary. 
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statistical bias called the modifiable areal unit problem (MAUP),^^ whieh renders sueh 
estimates highly unreliable. In eontrast, the approaeh taken here seeks to eonvert the 
metries derived from soeial media messages into eontinuous spatial surfaees. For eaeh 
eoneept and for eaeh month in our study period, we estimated a eontinuous spatial 
surfaee, representing the relative density of messages refereneing a eoneept in a partieular 
plaee and time. We smoothed the results using two-dimensional binned Gaussian kernel 
density interpolation.^^ Jq avoid arbitrary limitations on the spatial ranges of the 
estimated effeets, we generated densities by taking the average aeross multiple kernel 
density bandwidths, ranging from 1 kilometer to 500 kilometers. The result is a heat map 
representing the relative density of messages about a partieular sentiment and eoneept, as 
illustrated in Figure 1, whieh shows the negativity toward the National Government 


Yemen 

( 2014 - 01 - 01 ) 



Relative 
Point Density 

Hr 100 


- 10 


- 1 


- 0.1 



0.01 


Figure 1. Heat Map. National Government Negative Tweets. January 2014.^^ 


Stan Openshaw and S. Openshaw, The Modifiable Areal Unit Problem (Geo Abstracts University of 
East Anglia, 1984). 

T. Camber Warren, “Mapping the Rhetoric of Violence: Political Conflict Discourse and the 
Emergence of Identity Radicalization in Nigerian Social Media,” prepared for presentation at the Annual 
Meeting of the American Political Science Association, September 3-6, 2015, San Francisco, CA. 

Adapted from Twitter, unpublished data. 
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4. Dependent Variables 


To attempt to validate the inferenees derived from our new social media metrics, 
we rely on existing opinion poll data, collected by Gallup between November 14, 2013, 
and January 17, 2014, under the Global Research and Assessment Program (GRAP) for 
United States Special Operations Command (USSOCOM) J39. The poll consisted of 
9,450 civilian Yemeni adults aged 16-65 with a sampling based on a stratified, 
multistage cluster design. The key advantage of this data source is that it provides 
anonymous measurements of political support for the Yemeni state and various insurgent 
groups, which can be geo-referenced to specific cities and towns, allowing researchers to 
generate an aggregate spatial representation of the sentiments recorded by the poll.^^ 

All dependent variables were coded dichotomously. 

Extremism Support and Extremism Opposition are derived from the 
Extremism index, which ranges from one to four and is a compilation of 25 questions 
concerning violent extremist organizations (VEOs), attitudes toward sharia, and jihad, 
with a Cronbach’s alpha score of 0.62. 

Institutional Support and Institutional Opposition are derived from the 
Institutional Confidence index, which is a compilation of 10 questions with a Cronbach’s 
alpha score of 0.91. This index included questions about confidence in national 
government, state/provincial government, local/city government, military, police/law 
enforcement, financial institutions/banks, the national courts, sharia courts, tribal justice, 
and the electoral system. 

Al-Qaeda Support and Al-Qaeda Opposition counted any responses indicating 
that either al-Qaeda’s or AQAP’s influence had a very positive influence as support. 
Similarly, any responses indicating these organizations had a very negative influence 
were counted as an opposition. 


Jones, Counterinsurgency in Afghanistan. 
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National Government Support and National Government Opposition are 

derived by counting responses indicating confidence in the national government, with 
indications of no confidence used as indicator of opposition to the national government. 

5. Independent Variables 

The independent variables are based on the heat maps generated from the 
sentiment and concept dictionary results. They change depending upon the evaluated 
dependent variable. Additionally, each independent variable forms the basis of a model 
set. These sets are evaluated in three different ways with demographic control variables, 
social media variables, or a combination of the two. We annotate these as 
DEMOGRAPHIC VARIANCE, TWITTER VARIANCE, and COMBINATION 
VARIANCE. 

The independent variable extremism is the result of all words associated with 
VEOs and jihadist ideology as noted in the Sentiment Dictionary section. The 
independent variables are divided between either EXTREMISM POSITIVE, which are 
based on the positive sentiment dictionary, or EXTREMISM NEGATIVE, which is 
based on the negative sentiment dictionary. It is further broken down MONTHLY from 
October to January, and a TOTAL of all four months for a total of 10 extremism 
independent variables. Along the same lines each single organization was evaluated. Eor 
instance, in evaluating support for al-Qaeda, there is an AQ POSITIVE and an AQ 
NEGATIVE independent variable for each month, and a TOTAL of all four months for a 
total of 10 AQ independent variables. 

The independent variable of INSTITUTION is the result of all words associated 
with institutions noted in the Sentiment Dictionary section. The institution variables are 
divided like the extremism variables. The institution variables are divided into the 
variables INSTITUTION POSITIVE and INSTITUTION NEGATIVE. Insitution is 
further broken down into individual months, called MONTHLY from October to January 
with a TOTAL of all months for a combined 10 institution independent variables. Along 
the same line, the national government was evaluated. There is a NATIONAL 
GOVERNMENT POSITIVE and a NATIONAL GOVERNMENT NEGATIVE 
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independent variable for most months, and a TOTAL of all the months for a total of 9 
national government independent variables. There are fewer national government 
independent variables because there were not enough Twitter results to generate a 
positive heat map for the month of October for this search category. 

6. Control Variables 

Most of the control variables are based on demographic information. We did this 
to address a common critique against using social media data, which is that social media 
users are not a representative sampling of the population. Some may be concerned that if 
we find a significant relationship that appears to be driven by social media data, it may 
actually reflect demographic factors. Therefore, we controlled for a series of 
demographic factors that are known to influence political attitudes: Internet access, 
urban, gender, age, education, and wealth. All control variables were coded 
dichotomously. 

The other control variables are derived from the Twitter density estimates. We 
controlled for both the overall number of tweets and the total number of tweets about a 
concept in a specific timeframe. We did this to account for the influence of total message 
volume so it could be differentiated from message content (e.g., sentiment). We also 
controlled for 

Internet assigns a value of 1 to any use of the Internet, Urban is coded as I for 
urban areas with a population of at least 5,000 inhabitants. Youth counts those 24 years 
of age or younger. Educated counts those with at least some university education. This 
variable was dichotomously coded with those with at least some university education as 1 
and everyone else as 0. 

Wealthy - This control variable was derived from the Gallup Poll. It reflects 
Yemenis with an income of 60,000 Yemeni Riyals (YER) per year or higher. Anyone 
earning at least 60,000 YER was dichotomously coded as 1 and those below that amount 
as 0. 

We also control for background spatial characteristics: 
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Population Density - This control variable is derived from a population density 
heat map and refleets a more refined impact of population in the urban areas. 

Total Tweets - This eontrol variable is derived from the eombination of heat 
maps for eaeh month from Oetober 2013 through January 2014. It is the sum of all tweets 
originating from Yemen during this timeframe. This variable was log transformed in 
order to aeeount for its heavy tailed distribution. 

Total Concept Tweets - This eontrol variable is based on the eombination of heat 
maps that relate to eaeh of the independent variables. It eneompasses the months of 
Oetober 2013 to January 2014. It is the sum of all tweets that relate to a specifie topie 
(e.g., extremism, national government) during the speeified time. This variable was also 
log transformed in order to aeeount for its heavy tailed distribution. 

Provincial Boundaries - This control variable is based on the 20 provincial 
political boundaries and the capital city boundary within Yemen. These were included as 
fixed effects in the models. 

C. REGRESSION ANALYSIS 

We used a logistic regression for our analysis. Logistic regressions are a statistical 
method in which the dependent variable is dichotomous, also known as logit regressions, 
or the logit model. Dichotomous dependent variables can take only two values, such as 
one/zero, win/lose, or alive/dead. The logistic model is used to estimate the probability of 
a binary response based on one or more independent variables. 

In summary, we assessed eight dependent variables that were derived from existing 
opinion polls. The dependent variables focus on support for or opposition to the 
government and support for or opposition to the insurgency. After developing the 
dependent variables we built 39 independent variables that derived from positive and 
negative Twitter sentiment. Finally, we controlled for a variety of factors related to the 
dependent and independent variables. In the end, we experimented and evaluated 117 
different models. 

Cox, “The Regression Analysis of Binary Sequenees.” 
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V. RESULTS 


Taken as whole, these statistical models generate five major categories of results. 
First, we find that the addition of social media metrics consistently improves the level of 
predictive success when assessing popular sentiments, over and above the predictions 
generated by demographic patterns alone. Second, we find that using our method, we are 
able to generate estimates of spatial patterns of sentiment that strongly resemble estimates 
derived from traditional polling methods. Third, it seems that this congruence is 
strongest when our method was used to identify oppositional sentiments, as opposed to 
supportive sentiments. Fourth, our methods yield better and more consistent results when 
examining VEO’s and extremism, than when examining support for the government. 
Finally, when we examine data by month, even controlling for background spatial 
patterns and demographic patterns, we find that the social media data still provides an 
important improvement in the predictive success of the models. 

A. FINDING ONE - IMPROVES PREDICTIONS 

The most significant and reliable finding is that Twitter-derived sentiment metrics 
strengthens the predictive success of the statistical models. Every model with Twitter and 
demographic data combined resulted in better AIC scores than those models with either 
demographic or Twitter data by themselves. Why is this important? The AIC score is 
basically a comparison tool that helps us evaluate the predictive success of each model. 
The AIC measures how well a specific model fits a dataset, which in this case are 
estimates of political support and opposition. It also considers the number of variables, 
penalizing those models that are so flexible that can just as easily fit any dataset. Most 
importantly the AIC score is an indicator of confidence in a model. 

When examining the findings presented in Table I, it is important to note that the 
lower the AIC score is, the more confidence we should have in the model. These eight 
model sets are those with the TOTAL independent variables. We believe that these 

Hirotugu Akaike, “A new look at the statistical model identification.” IEEE transactions on 
automatic control 19, no. 6 (1974). 


27 



particular models are the most reliable, beeause they encompass Twitter data for the same 
time period that the original opinion poll was conducted. The results on the far right show 
the AIC scores for each of the combined models, supporting our claim of improved 
predictive sueeess aeross the board. 


Table 1. Akaike Information Criterion. 


DEPENDENT VARIABLES | 

Demographics 

Twitter 

Combined 

A1 Qaeda Support " 

612.572 

649.248 

541.813 

Al Qaeda Opposition 

9,423.769 

9,700.528 

8,134.125 

Extremism Support 

5,144.949 

5,387.780 

4,299.991 

Extremism Opposition 

3,103.926 

3,483.122 

2,857.858 

National Govt Support 

6,592.986 

6,443.039 

5,764.233 

National GoaT Opposition 

9,479.310 

11,007.990 

8,754.131 

Institution Support 

8,459.854 

8,322.456 

7,297.280 

Institution Opposition 

9,914.237 

10,735.100 

8,576.029 


B. FINDING TWO - SIMILARITY OF SPATIAL PATTERNS 

Closely related to the first finding is that the estimates derived from social media 
generate similar patterns of inferences to those generated by traditional polling data. 
There are two ways to demonstrate this finding. The first is a statistical approach and the 
other is a visual depiction. Statistically, 30 out of the 39 TWITTER VARIANCE models 
show a better AIC than the DEMOGRAPHIC VARIANCE models. The only model sets 
that do not show that the TWITTER VARIANCE is better are those with National 
Government Support and Institutional Support dependent variables. 

The other way to demonstrate this is through the visual depiction in Eigure 2, 
which shows a comparison of polling and twitter sentiment by provinee. The heat map on 
the left is the mean of all polling responses eoneeming AQ and AQAP by province. The 
darker the eolor means there is a greater the level of opposition. The heat map on the 
right is the mean of all negative tweets about AQ and AQAP from Oetober to January by 
province. The darker color is there is a greater amount of negative tweets. As you ean see 
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from the maps, the majority of the provinces are closely aligned with the exception of 
eastern Yemen, an extremely rural region with little to no Internet access. 


Can social media alg:orithms 
replicate survey results? 


Gallup Survey Response 


‘Are you opposed to al-Gaeda?” 


Twitter Data Algorithm Result 


“Are you opposed to al-Qaeda?’' 


N 
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Authors; Andrew Bourret. Joshua Wines, Jason Mendes 


Figure 2. Comparing Survey Responses and Twitter Metrics Indicating 

Opposition to al-Qaeda.^* 


This is not to assert that our research finds that twitter sentiment can be used in 
lieu of polling. There are too many inconsistencies between the different models to 
definitively make such a claim. The purpose is to highlight the competitiveness between 
the impact of twitter data and demographic data on polling sentiment. It is simply another 
step in confirming or denying the value of social media as a source of data-driven 
insights. 


Adapted from Gallup survey, unpublished data; Twitter, unpublished data. 
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c. 


FINDING THREE - SENTIMENT MATTERS 


Interestingly, our results also indieate that Twitter is more suitable for pieking up 
certain types of information. For example, the results in Table 2 indicate that negative 
sentiment may be more reliable. These results are the only TOTAL independent variables 
assessed using the Combination Variance that had statistically significant coefficients. 
Just as importantly, they show a positive relationship between polling and twitter 
sentiment. As mentioned earlier in Finding One, we have the greatest confidence in the 
TOTAL independent variables and the Combination Variance always has the best AIC. 
These relationships can be found looking at 1) the AQ Negative Sentiment row under the 
al-Qaeda Opposition combined column; 2) the Extremism Negative sentiment row under 
the Extremism Opposition combined column. 


Table 2. Combined Negative Sentiment. 


Al 

Demographies 


Qaeda Opposition 

Inittcr C'ombined 


Extremism Opposition 

Demographies Twitter Combined 


Internet Users 

0.517*" (0.122) 

0.254* (0.141) -0.127(0.201) 


-0.158(0.246) 

Urban Residents 

0.038 (0.073) 

0.148* (0.086) -0.062(0.146) 


-0.033(0.167) 

Men 

1.226*" (0.054) 

1.271*** (0.058) -0.564*** (0.102) 


-0.750*** (0.117) 

Youth 

0.087 (0.056) 

0.052(0.061) 0.259** (0.104) 


0.176(0.116) 

Educated 

0.537*** (0.111) 

0.605*** (0.125) 0.047(0.162) 


0.242 (0.183) 

Wealthy 

0.391*** (0.066) 

0.386*** (0.073) 0.138(0.121) 


0.083(0.1.54) 

Population Density 

0.016(0.0.54) 

-0.022 (0.039) -0.080 (0.063) 


-0.120 (0.077) 

AQ Ne§ati\ e Sentiment 

2.012*** 

(0.235) 0.799*** (0.305) 



Total AQ Tw eets 

-1.281*** 

(0.328)-1.089** (0.428) 



Extremism Negative Sentiment 


2.405** 

(0.332) 

0.822* (0.486) 

l olal I'‘xtremism Tweets 


-2.409* 

(0.632) 

0.208 (0.753) 

Fotal I w eels 

-0.132*" 

(0.036) -0.027(0.062) -0.406* 

(0.084) 

-0.093 (0.148) 

Constant 

-0.191 (0.227) -1.764*** 

(0.408) 0.163 (0.599) -2.535*** (0.446)-5.260* 

(1.0.50)-4.759*** (1.541) 


Province fixed etTccts 
AIC 


NA 

9700.528 


Included 

942.r.769 


Included 

81.U.125 


NA 

.548.1.122 


Included 

.510.5.926 


Included 

2857.858 


p < .01; p < .05; p < .1 


Columns are comprised of two values. Numbers on the left side of each column are coefficient 
values. Numbers to the right side of each column are standard error. 


Even in the negative Total independent variables with Twitter Variance, only one 
model with a statistically significant coefficient showed a negative relationship (Table 3). 
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This reinforces the finding that sentiment is potentially better at evaluating negative 
sentiment. 


Table 3. Significance in Expected Direction (Total). 


DEPENDENT VARIABLES | 

Expected 

Unexpected 

1 

Insignificant 

Al Qaeda Support ^ 

X 



Al Qaeda Opposition 

X 

Extremism Support 

X 


Extremism Opposition 

X 

National CoxT Support 


X 


National Govt Opposition 

X 

Institution Support 

X 


Institution Opposition 

X 


When we looked at the reliability of positive sentiment, we found some additional 
supporting arguments. For example, there was only one other TOTAL independent 
variable model we assessed using the COMBINATION VARIANCE that had statistically 
significant results. This was the National Government Support model, but it had a 
negative relationship between twitter and polling (reference Table 4). This result along 
with the negative relationship found in the Institutional Support TWITTER VARIANCE 
model seems to reinforce the finding that negative sentiment is more reliable. We infer 
that Twitter data is a reliable source of ground truth when there is a positive relationship 
between positive twitter sentiment / polling support and negative twitter sentiment / 
polling opposition. Based on this analysis, it seems like those that spoke negatively on 
twitter were likely to be in the same geographic space as those that spoke negatively in 
the survey. 
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Table 4. Combined Sentiment. 


National Government Support Institutional Support 



Demographics 

Twitter 

Combined 

Demographics 

Twitter 

Combined 

Internet Users 

-o..r.r8”(0.i58) 


-0.419” (0.175) 

-0.205 (0.125) 



-0.284” (0.144) 

Urban Residents 

-0.292‘''(0.101) 


-0.275” (0.114) 

-0.169” (0.082) 



-0.161'(0.092) 

Men 

-0.277''' (0.067) 


-0,.4.47'''(0.071) 

-0.602'” (0.057) 



-0.612'” (0,061) 

Youth 

0.059 (0.074) 


0.098 (0.077) 

0,121'(0.062) 



0.186'” (0.065) 

I'.dueated 

-0.166(0.129) 


-0.145 (0.147) 

-0.247” (0,109) 



-0.208' (0.118) 

Wealthy 

0.188” (0.080) 


0.199” (0.086) 

0.049 (0.070) 



-0.002 (0.075) 

Population Density 

0.019(0.045) 


0.086 (0.054) 

-0.057 (0.048) 



-0.024 (0.044) 

Nall GoM Positive Sentiment 


-0.665”'(0.102) 

-0.491'” (0.145) 





Total Natl Go\ t Tweets 


0.442'” (0.140) 

0.247 (0.254) 





Institution Positiv e Sentiment 





-1.424”' 

■(0.451) 

-0.056 (0.444) 

Total Institution Tweets 





0.694” 

(0.448) 

-0.096 (0.475) 

Total Tweets 


0.077' (0.046) 

-0.249'” (0.087) 


0.451'” 

(0.045) 

-0.174'” (0.060) 

Constant 

-2.271'” (0..U5) 

-0.729' (0.492) 

-4.446'” (0.797) 

-0.752'” (0.254) 

2.546'” 

(0.290) 

-1..444” (0.570) 

Province fixed elTects 

NA 

Included 

Included 

NA 

Included 

Included 

AIC 

644.4.0.49 

6592.986 

5764,244 

8422.456 

8459 

1.854 

7297.280 


p < .01; p < .05; p < .1 


Columns are comprised of two values. Numbers on the left side of each column are coefficient 
values. Numbers to the right side of each column are standard error. 


If true, this may mean that the negative sentiment dietionary is perhaps better due 
to translation. However, previous researeh shows that sentiment dictionaries that are 
machine translated perform only slightly worse than manually created dictionaries.^^ 
other possible explanation is that Yemeni Twitter users that express negative sentiment 
are a more representative sample of the population. This would be consistent with other 
research that states social media in the Middle East functions as a “free space” for 
criticism. 

D. FINDING FOUR - THE IMPORTANCE OF THE TOPIC 

The story does not end there. Some of the other results challenge the third finding. 
For example, every extremism and AQ related model with MONTHLY independent 
variables (e.g. October, November, etc.) using the COMBINATION VARIANCE or 
TWITTER VARIANCE results in statistically significant coefficients and a positive 
relationship. This is regardless of whether the independent variable is based on either 

SaifM. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko, "How Translation Alters 
Sentiment" Journal of Artificial Intelligence Research 1 (2015): 95-130. 
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positive or negative sentiment. In contrast, only the National Government Support and 
the Institutional Support models have relationships in both the expected and unexpected 
direction. This contrast is seen in Table 5, which distills the results of all of the 
MONTHLY independent variables using either the COMBINATION VARIANCE or 
TWITTER VARIANCE. It captures any MONTHLY statistically significant coefficient 
that has a positive or negative relationship between twitter and polling. As you can tell, it 
seems that certain concepts are easier or more feasible to evaluate. 


Table 5. Significance in Expected Direction (Any). 


DEPENDENT VARIABLES 

Expected 

Unexpected 

Al Qaeda Support 

X 


A1 Qaeda Opposition 

X 

Extremism Support 

X 


Extremism Opposition 

X 

National Govt Support 


X 

National Cost Opposition 

X 

X 

Institution Support 

X 

X 

Institution Opposition 

X 

X 


These inconsistencies point to the possibility that the dictionary didn’t adequately 
capture the concepts of national government and/or the institution index. However, there 
were 51 separate terms associated with national government and 890 separate terms 
associated with the institution index. It’s unlikely the inconsistencies are due to a 
deficient dictionary. It’s also possible that individuals that tweet negatively about the 
government are less likely to self-identify their location in their profile out of fear. 
However, there are negative relationships even among those that tweet positively about 
the government and positive relationships among those that tweet positively about AQ. 
Tear of the government does not seem to fit as an answer. It may simply be that it’s more 
difficult to evaluate concepts regarding the government compared to AQ and extremism. 
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E. 


FINDING FIVE - POTENTIAL TO EVALUATE BY MONTH 


Lastly, we evaluated each dependent variable using the MONTHLY positive and 
negative independent variables. This was the case for each model except for NATIONAL 
GOVERNMENT POSITIVE sentiment. As noted earlier, there weren’t enough Twitter 
data points to make a heat map for the month of October. The value of running so many 
models with different independent variables allowed us to examine the reliability of 
monthly data. In fact, we looked at 35 model sets of MONTHLY independent variables. 

These models indicate that monthly Twitter sentiment may also be reliable. We 
stated in Finding One that we have the greatest confidence in the TOTAL independent 
variables. However, even the majority of the MONTHLY independent variable produces 
results with statistically significant coefficients in the expected direction. Table 6 
captures the number of the MONTHLY TWITTER VARIANCE models with statistically 
significant coefficients. Out of those 27 models, 18 of them show a positive relationship. 
These statistical outputs are important, because they influence our confidence level in the 
reliability of the data. While these results aren’t as statistically strong as the previous 
findings, they do indicate there is a significant relationship between monthly Twitter data 
and polling results. 


Table 6. Significance in Expected Direction (Monthly). 


DEPENDENT VARIABLES 


October 

November 

December 

Januars' 

Al Qaeda Support 

“I 

Expected 

Expected 


Al Qaeda Opposition 

Expected 

Expected 

Expected 


Extremism Support 

Expected 

Expected 

Expected 

Expected 

Extremism Opposition 

Expected 


Expected 

Expected 

National Govt Support 

Unexpected 

Unexpected 

Unexpected 

Unexpected 

National Govt Opposition 

Unexpected 

Expected 

Unexpected 

Expected 

Institution Support 

Expected 

Expected 

Unexpected 


Institution Opposition 

Expected 

Unexpected 

Unexpected 

Expected 
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VI. CONCLUSION 


Our thesis asked whether soeial media analysis eould be a reliable proxy for 
polling in a eonfliet zone. We decided to pursue this research question after identifying 
several gaps. First, we examined the importance of traditional collection and assessment 
methods in determining a population’s attitudes, beliefs, and opinions, specifically in 
low-intensity conflicts. During our research, we identified a deficiency in the U.S. 
military’s doctrinal assessment methods. Primarily, we noted a need to evaluate popular 
support more often than traditional methods afford, especially in areas outside major 
combat operations. 

This subsequently led us to examine different ways social media data is being 
used, and we identified sentiment analysis as the most appropriate way to evaluate this 
data. However, there was one specific aspect missing from the existing literature: the 
assessment of the relationship between geo-spatially anchored social media users and 
traditional polling methods. In response, we designed an approach centered around 
examining geo-spatially anchored data, allowing us to assess whether people in the same 
general location express the same sentiment in both measurement approaches. 

During the analysis phase of our research, we examined dependent variables 
derived from existing polling data, alongside newly developed independent variables 
derived from Twitter messages. What we found in response to our question and research 
method was something unexpected—sentiment analysis of Twitter data has an additive 
positive value when compared to polling data, and metrics derived from Twitter 
messages may offer a reliable source of data under certain conditions. 
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VII. ADDITIONAL RESEARCH 


Our team identified two broad areas of potential future research. They are 
sentiment research and operational changes. The following recommended research areas 
are not limited in value to the U.S. government. 

A. IMPACT OF SENTIMENT 

Additional research is required to find out whether sentiment or topic selection 
under different conditions produces more reliable results as noted in this research. We 
believe it would be beneficial to determine whether sentiment dictionaries other than the 
ones we chose would provide different research outcomes. In particular, we recommend 
additional research to determine the impact of sentiment dictionary lists. We recommend 
using the same data and approach to establish a baseline in order to compare as many lists 
as are available. We also recommend developing AFINN dictionaries, at least for all 
high-interest areas. An Arabic AFINN would have been immeasurably helpful for this 
research and would have eliminated the need for translation. We recognize the potential 
negative impact of translation, so further research is also required to improve automated 
translation. This is especially true for the languages spoken in conflict areas and/or high- 
interest areas. 

B. EVALUATION OF THE TOPIC 

Our research indicates that the reliability and relationship of sentiment results 
vary by topic. However, only four topics were evaluated. What is needed is a deeper 
examination of which topics generate more reliable results using this data and data for 
other countries. This will help analysts identify gaps in social media data reliability. 

C. DOES THE COUNTRY/REGION MATTER? 

We believe it is important to determine whether our research method can be 
duplicated in other countries or regions. In particular, are there different outcomes for 
countries that are at peace versus those in conflict? 
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D. GEO-LOCATION 

One explanation for the statistieally insignificant results and inverse relationships 
found in our research is that there simply were not enough tweets in Yemen. Increasing 
the number of available geo-located data points would improve the reliability of our 
research. Additionally, more data could help shed light on the role online obfuscation 
plays. When online users obscure their location or identity, does this change the content 
of their tweets? Are they more or less honest? Does this impact the sentiment and focus 
of criticism or praise for the government or insurgent groups? Along the same lines, the 
USG should require latitude and longitude for all future contracted and executed polls in 
order to conduct robust geo-spatial analyses. 

E. INTEGRATION OF SOCIAL MEDIA ANALYSIS 

As noted at the beginning of the chapter on results, social media data combined 
with polling data produces a better and more reliable model of analysis. Analysts should 
conduct social media analysis and combine the results with polling analysis to support 
operational assessments. 
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APPENDIX. CONCEPT DICTIONARY 


Extremism & 
AQAP 

Extremism & 

AO 

Extremism & 

AS 

Extremism & 

AA 

AQAP 

AO 

Ansar al Shariah 

Ansar Allah 

AQ in Yemen 

al-Qaeda 

AAS 

Houthi militants 

al-Qaeda in Yemen 

al zawahiri 

ASY 

Houthi fighters 

al-Qaeda in the 
Arabian Peninsula 


Hamza al Zinjibari 

Houthi rebels 

Qasim al-Raymi 


Jalal Baleedi 

Houthi militia 

Sheikh al-Raymi 


Hamza al-Zinjibari 

Yahia Badreddin al- 
Houthi 

Emir al-Raymi 



Abdul Malik 
Badreddin 

Nasir al-Wuhayshi 



Abu Ali Abdullah al 
Hakem 

Sheikh al-Wuhayshi 




Emir al-Wuhayshi 





Extremism & 
Jihad 

Extremism & 
Sharia 



Aden-Abyan 

Sharia 



Islamie Army 

Islamic Court 



Abdul-Karim 




Jihad 




Wahhabi 




Salafi 





Institution & 
Military 

Institution & 
Military cont’d 

Institution & 
Police 

Institution & 
National govt 

Yemeni Military 

Yemeni Air Eorce 

Eaw enforcement 

ROYG 

Our Military 

Y emeni Airmen 

Central Security 

Government 

Our Military 
Eeadership 

Yemeni Airman 

Police 

President 

Yemeni military 
leaders 

Our Air Eorce 

Security Eorces 

Prime Minister 

Military Council 

Our Airmen 

Government forces 

Parliament 

Military headers 

Yemeni Eighter 

Jets 

Political Security 
Organization 

Yemeni Council 

Yemeni Security 

Eorces 

Our Eighter Jets 

National Security 
Bureau 

Yemeni Regime 

Yemeni Armed 

Yemeni Navy 

Counter-Terrorism 

Republic of 
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Forces 


Unit 

Yemen 

Our Security Forces 

Our Navy 

Ministry of Interior 

Political headers 

Our Armed Forces 

Our Sailors 

Minister of Interior 


Our Troops 

Yemeni 

Battleship 

Criminal 

Investigation 


Yemeni Troops 

Our Forces 

Abdel-Qader Qahtan 


Yemeni Army 

Our Government 
Forces 

Abdel Qader Qahtan 


Our Army 

Yemeni Forces 

CSF 


Our Soldiers 

Yemeni 

Government 

Forces 

NSB 


Yemeni Soldiers 

Yemeni 

Republican Guard 

CTU 


Our Republican 

Guard 

The Ministry of 
Defense in Aden 



Yemeni Ministry of 
Defense 

Our Ministry of 
Defense 



Yemeni Minister of 
Defense 

Mohammed 

Nasser Ahmed 




Institution & 
Local govt 

Institution & 
National govt 

Institution & 
Federal Court 

Institution & 
Tribal Justice 

Focal government 

ROYG 

Justice System 

Tribal justice 

City government 

Government 

Court 

Tribal court 

Mayor 

President 

Judges 


City Manager 

Prime Minister 



City Council 

Parliament 



My Municipality 

Yemeni Council 



Municipal council 

Yemeni Regime 




Institution & 
Bank 

Institution & 
Elections 

Institution & 

State govt 


Yemeni bank 

Electoral 

Governor 


SBYB 

Election 




Notes: The Extremism annotation in the first table indicates that all these terms are 
associated with the concept of extremism in addition to the 6 subcategories of extremism. 
The Institution annotation in the second table indicates that all these terms are associated 
with the concept of institution in addition to the 10 subcategories of Yemeni institutions. 
This is not an exhaustive list with spelling and punctuation variations removed along with 
the Arabic translation. 
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